dbt Coalesce 2024: Unifying the Data Ecosystem with Data Quality at Center Stage

Collaboration, Democratization, and Data Quality in the Time of the Data Control Plane

Data is a team sport. For 15 years I’ve built software to help technologists work and collaborate more effectively. In the data world, dbt Labs stands out as a leader in helping analytics engineers work smarter as a team.

Last week, I had the pleasure of speaking on behalf of Anomalo at dbt Coalesce 2024. As the leading transformation toolkit for cloud data analytics, dbt brings to data operations (“DataOps”) the same code-first principles that made software development and IT operations (“DevOps”) so successful: teams use dbt to turn raw data into analysis-ready insights every minute of every day, globally.

What excited me most about Coalesce 2024 is dbt’s focus on connecting the dots between data transformation, pipeline orchestration, data quality, and standardized business metrics (the “Semantic layer”) into dbt Cloud’s data control plane. Not only is dbt presenting a more complete vision for how these concerns fit together, but announcements like the new visual editor also show that dbt understands the need to bring analytics engineers and analysts together on a shared platform. 

As a data quality partner of dbt Cloud since 2022, we’re thrilled to hear this. Data quality in the enterprise has always been a data analyst problem; analysts are the ones impacted by quality issues and Anomalo has always focused on ease of use for analysts and business users. With Anomalo’s track record of innovation in data quality, and simple integration into dbt Cloud, it’s now easier than ever to bring enterprise-scale deep data quality to modern data teams.

For enterprise business leaders and data practitioners alike, here are my top takeaways from dbt’s fifth Coalesce.

The Data Control Plane: A New Paradigm for Enterprise Analytics

Kicking off Day 1 of Coalesce, dbt introduced the concept of the “data control plane” as a framework for orchestrating and observing entire data ecosystems. This paradigm shift, introduced as One dbt, aims to create a unified environment that works across clouds, data platforms, teams, and personas.

Core aspects of the improved data control plane include:

1. Cross-Platform Integration:
   • Cross-platform mesh capabilities including support for technologies like Apache Iceberg and flexibility across warehouses (e.g., Redshift, Snowflake, Databricks)
   • Support for cross-warehouse model references in dbt Cloud means more code reuse and clean logical separation of models between business domains
   • Supported by Anomalo’s platform-agnostic data quality monitoring, regardless of where you ingest, transform, analyze, or store your data
2. Enterprise-Scale Solutions:
   • Advanced CI features for continuous validation and documentation in complex data environments
   • Enterprise-grade solutions like Data Health Tiles and model-level notifications to share quality context more broadly
3. Unified Experience:
   • Interoperability of dbt Core and dbt Cloud helps companies adopt Cloud where it makes sense without requiring all-or-nothing migrations 
   • Emphasis on seamless workflows across different data environments

The data control plane serves as the foundation for enabling collaboration, democratization, and enhanced data quality in enterprise settings by linking four core capabilities:

• Data transformation
• Pipeline orchestration
• Catalog and metadata
• Quality

Collaboration and Democratization: Empowering Diverse Data Stakeholders

Using the framework of the data control plane, dbt is advocating for greater collaboration and democratization in data teams. This approach aims to foster shared ownership and accountability among a wider range of stakeholders in the data lifecycle–from data engineers to analytics engineers to business analysts and data consumers.

Key capabilities for democratization include:

1. Visual Editor:
   • Drag-and-drop interface for building dbt models
   • Empowers non-technical users and business analysts to contribute to data transformations
2. Semantic Layer Advancements:
   • Expanded integrations with BI tools (e.g., Tableau, Power BI) to ensure metric consistency across tools and teams, fostering a common data language
   • Anomalo’s automated anomaly detection provides predictive Machine Learning to  validate semantic-layer metrics for consistency and accuracy
3. Cross-Team Collaboration Features:
   • Best practices for data analysts to reconcile legacy SQL aggregations against semantic layer metrics
   • Support for data mesh architectures, facilitating domain-oriented data ownership

These features help break down silos between technical and business teams, enabling greater collaboration from data ingestion to insights.

Data Quality: The Cornerstone of Trustworthy Analytics

As organizations democratize access to data and analytics, ensuring data quality becomes paramount. AWS 2024 CIO Insights named data quality as the biggest blocker to data initiatives, nominated by 46% of data leaders. Throughout the conference, speakers and attendees alike stressed the importance of data quality in the success of their teams and Gen AI investments. 

Key developments in data quality include:

1. Integrated Quality Checks:
   • Companies like Roche are embedding dbt data and unit tests directly into their CI process as a best practice
2. Contextual Quality Signals:
   • Fox Corporation and other enterprises are making heavy investments in lineage tooling to assist in root cause analysis (RCA) and impact assessment. Adopting best-of-breed catalog products to serve as a “catalog of catalogs” across disparate warehouses and metadata stores is becoming more common in the enterprise.  
3. Deep (and broad) Data Quality:
   • Emphasis on proactive analysis and anomaly detection throughout the data processing pipeline to catch and resolve issues more quickly, and avoid the time and expense of running transforms on bad data
   • A growing frustration with rule-based approaches. Maybe data teams are maturing, or maybe more enterprise-scale teams were at Coalesce this year. I heard a lot of despair from people who are trying to scale rule-based data quality solutions past the breaking point.
   • Anomalo’s approach of using unsupervised machine learning (ML) to enhance the data control plane with deep, automated data quality checks resonated with a lot of practitioners and leaders I talked to.

These trends show that data teams are leveling up and getting more sophisticated, and tools are evolving to support them. Building enterprise-wide trust in data requires scalable practices, streamlined and integrated tooling, and bridge-building between traditional data teams and their internal customers across the company.

Opportunities and Future Direction

While the data control plane offers a compelling vision for enterprise analytics, the 2,000 live and 8,000 virtual attendees at Coalesce 2024 also raised many real challenges:

1. Balancing Self-Service and Governance: Finding the right balance between empowering business users and maintaining robust data governance, efficiency, and quality.
2. Alert Fatigue: Addressing the issue of noisy, non-personalized alerts as data quality monitoring scales.
3. Performance Optimization: Ensuring comprehensive quality checks don’t negatively impact pipeline performance.
4. Skill Development: Supporting the transition of data professionals and analysts to more collaborative, quality-focused workflows.

Ultimately, dbt’s vision for the data control plane goes beyond technology: it’s about empowering people across organizations to work more effectively with data. As we strive for democratization, data quality and governance remains crucial. The focus on integrated quality checks resonates strongly with Anomalo’s mission to ensure data quality at scale. 

For leaders and practitioners alike, the message is clear: siloed, opaque data processes are becoming obsolete. The future lies in collaborative, transparent, and quality-centric data ecosystems. Embracing this paradigm will unlock unprecedented value from data assets. 

If these challenges and solutions resonate with you, you can learn more about Anomalo’s dbt integration or request a demo. See you at Coalesce 2025!